Machine Learning Engineer Nanodegree

Capstone Project

Landmark Recognition

Fady Morris Milad Ebeid
December 5, 2019

I. Definition

Project Overview

Computer vision algorithms include methods for acquiring, processing, analyzing and understanding digital images, and extraction of data from the real world. It is an interdisciplinary field that deals with how can computers gain a high-level understanding of digital images. It aims to mimic human vision.
Convolutional neural networks are now capable of outperforming humans on some computer vision tasks, such as classifying images.
In this project, I provide a solution to the Landmark Recognition Problem. Given an input photo of a place anywhere around the world, the computer can recognize and label the landmark in which this image was taken.
The dataset I used for this project is Kaggle Google Landmark Recognition 2019 dataset and can be downloaded from Common Visual Data Foundation Google Landmarks Dataset v2.

Importing Libraries :

In [1]:
import numpy as np
import pandas as pd
from IPython.display import display # Allows the use of display() for DataFrames
import matplotlib.pyplot as plt

# Pretty display for notebooks
%matplotlib inline
In [2]:
import sys, os
from os import path
import csv
import pickle #object binary serialization
In [3]:
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' #suppress tensorflow warnings
In [4]:
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense, Conv2D, MaxPooling2D, GlobalAveragePooling2D
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.utils.np_utils import to_categorical
from keras.utils import plot_model
from keras import applications, optimizers
from keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img
Using TensorFlow backend.

Connecting to google drive and copying dataset to colab virtual machine :

In [0]:
from google.colab import drive
drive.mount('/gdrive')
In [0]:
cd /
/
In [0]:
%%shell
cp -v /gdrive/My\ Drive/shared/Udacity_MLND_capstone_dataset/data.tar.gz /
tar xvzf data.tar.gz
mkdir models
mkdir -p docs/figures
mkdir -p docs/stats

Data paths definitions :

In [5]:
# Data paths definitions :
data_dir = "../data"
data_processed_dir = path.join(data_dir, "processed")
train_dir = path.join(data_processed_dir, "train") ##Training images directory
validation_dir = path.join(data_processed_dir, "validation") ##Validation images directory
test_dir = path.join(data_processed_dir, "test") ##Validation images directory

test_images_dir = path.join(data_dir, "test_images") #test images directory (unlabeled images used for prediction)

models_dir = "../models"
docs_dir = "../docs"
stats_dir = path.join(docs_dir, "stats") #Output csv statistics directory
figures_dir = path.join(docs_dir, "figures") #Output directory for saving figures

Problem Statement

Did you ever go through your vacation photos and ask yourself: What is the name of this temple I visited in China? Who created this monument I saw in France? Landmark recognition can help! This technology can predict landmark labels directly from image pixels, to help people better understand and organize their photo collections.
This problem was inspired by Google Landmark Recognition 2019 Challenge on Kaggle.
Landmark recognition is a little different from other classification problems. It contains a much larger number of classes (there are a total of 15K classes in this challenge), and the number of training examples per class may not be very large. Landmark recognition is challenging in its way.
This problem is a multi-class classification problem. In this problem, I built a classifier that can be trained using the given dataset and can be used to predict the landmark class from a given input image.
I have chosen to use convolutional neural networks and transfer learning techniques as classifiers. CNNs yield better results than traditional computer vision algorithms. I trained a basic convolutional neural network, then used pre-trained VGG16 and Xception models in transfer learning to solve the Google Landmark Recognition 2019 Problem.

Metrics

I will use two evaluation metrics for my project. Accuracy score, which is implemented in Keras, and Global Average Precision Metric (GAP).

Accuracy Score

The primary evaluation metric for this problem is the accuracy score. Since our dataset is balanced (not skewed), the accuracy score can be used successfully. Accuracy is the fraction of predictions our model got right. $$\text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$$

$$\texttt{accuracy}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i = y_i)$$

Global Average Precision Metric

This metric from Google Landmark Recognition 2019 - Evaluation Metric} is also known as micro Average Precision (microAP), as per F. Perronnin, Y. Liu, and J.-M. Renders, "A Family of Contextual Measures of Similarity between Distributions with Application to Image Retrieval," Proc. CVPR'09 works as follows:

  • For each query image (data point) in the input of $N$ images (submission), predict landmark label (class) and corresponding confidence score (probability).
  • Sort data points in the list of predictions in descending order by their confidence scores.
  • The Global Average Precision is computed as : $$GAP = \frac{1}{M}\sum_{i=1}^N P(i) rel(i)$$

where:

  • $N$ is the total number of predictions returned by the system, across all queries
  • $M$ is the total number of queries with at least one landmark from the training set visible in it (note that some queries may not depict landmarks)
  • $rel(i)$ denotes the relevance of prediction $i$ $$ rel(i)= \begin{cases} 1, \qquad \text{if the $i$-th prediction is correct}\\ 0, \qquad \text{otherwise} \end{cases} $$
  • $P(i)$ is the precision at rank $i$, $$\displaystyle P(i) = \frac{1}{i}\sum_{j=1}^i rel(j)$$ GAP score favors the correct predictions with higher confidence over the correct predictions at lower confidence.

The implementation of this metric was obtained from David Thaler GAP metric implementation on Kaggle.

In [6]:
# Script Source : [Kaggle - David Thaler - Gap Metric](https://www.kaggle.com/davidthaler/gap-metric)
def GAP_vector(pred, conf, true, return_x=False):
    '''
    Compute Global Average Precision (aka micro AP), the metric for the
    Google Landmark Recognition competition. 
    This function takes predictions, labels and confidence scores as vectors.
    In both predictions and ground-truth, use None/np.nan for "no label".

    Args:
        pred: vector of integer-coded predictions
        conf: vector of probability or confidence scores for pred
        true: vector of integer-coded labels for ground truth
        return_x: also return the data frame used in the calculation

    Returns:
        GAP score
    '''
    x = pd.DataFrame({'pred': pred, 'conf': conf, 'true': true})
    x.sort_values('conf', ascending=False, inplace=True, na_position='last')
    x['correct'] = (x.true == x.pred).astype(int)
    x['prec_k'] = x.correct.cumsum() / (np.arange(len(x)) + 1)
    x['term'] = x.prec_k * x.correct
    gap = x.term.sum() / x.true.count()
    if return_x:
        return gap, x
    else:
        return gap

Here We define a class that records performance metrics to csv file and initialize an instance of it:

In [7]:
class StatsCSV:
    def __init__(self, csv_file):
        self.csv_file = csv_file
        with open(self.csv_file, 'w', newline='') as csvfile:
            header_writer = csv.writer(csvfile)
            header_writer.writerow(['Model', 'Test Loss', 'Test Accuracy', 'Test GAP'])
            
    def add_stats(self,model_name, loss, accuracy, GAP):
        with open(self.csv_file, 'a', newline='') as csvfile:
            stats_writer = csv.writer(csvfile)
            stats_writer.writerow([model_name, round(loss,4), round(accuracy * 100, 2), round(GAP * 100, 2)])
In [0]:
stats_csv = StatsCSV(path.join(stats_dir, "stats.csv"))

II. Analysis

Data Exploration

Variables and Class definitions :

In [8]:
class IndexCSV:
    def __init__(self, name, index_csv_dir, index_csv_file):
        try:
            self.index_data = pd.read_csv(path.join(index_csv_dir, index_csv_file), usecols=['id', 'category'])
            print("File has {} samples with {} features each.".format(*self.index_data.shape))
        except:
            print("File could not be loaded. Is the dataset missing?")
        self.name = name
        self.fig = None
        self.ax = None
        self.freqs = None

    def get_freqs(self):
        self.freqs = self.index_data['category'].value_counts().to_frame()
        self.freqs.columns = ['images_count']
        return self.freqs
    def get_plot(self):
        #self.fig, self.ax = plt.subplots()
        self.ax = self.index_data['category'].value_counts().plot.barh(title=self.name)
        self.ax.set(xlabel='Images Count', ylabel='Landmark (Class) Name')
        self.ax.grid(True)
        return self.ax

Load index csv files :

In [9]:
index_train_csv = IndexCSV("Training Data", data_processed_dir, "index_train.csv")
index_validation_csv = IndexCSV("Validation Data", data_processed_dir, "index_validation.csv")
index_test_csv = IndexCSV("Test Data", data_processed_dir, "index_test.csv")
File has 7083 samples with 2 features each.
File has 1518 samples with 2 features each.
File has 1518 samples with 2 features each.

Selected Landmarks (Categories)

In [10]:
landmarks_list = index_validation_csv.index_data.groupby('category').nunique().index.to_list()
landmarks_df = pd.DataFrame(landmarks_list , index=np.arange(1, len(landmarks_list) + 1), columns = ['Landmarks'])
display(landmarks_df)
Landmarks
1 Burrator
2 Dead_Sea
3 Feroz_Shah_Kotla
4 Golden_Gate_Bridge
5 Hayravank_monastery
6 Kasteel_Amerongen
7 Kazan
8 Masada
9 Matka_Canyon
10 Mount_Arapiles
In [11]:
landmarks_df.to_csv(os.path.join(stats_dir, "selected_landmarks.csv"), index=True) #Save to CSV

Training Data Frequencies and Count :

In [12]:
train_freqs = index_train_csv.get_freqs()
display(train_freqs)
images_count
Masada 739
Dead_Sea 734
Hayravank_monastery 730
Golden_Gate_Bridge 726
Mount_Arapiles 719
Matka_Canyon 712
Feroz_Shah_Kotla 685
Burrator 683
Kazan 682
Kasteel_Amerongen 673
In [13]:
train_freqs.to_csv(os.path.join(stats_dir, "train_freqs.csv"), index=True)

Validation Data Frequencies and Count :

In [14]:
validation_freqs = index_validation_csv.get_freqs()
display(validation_freqs)
images_count
Masada 159
Hayravank_monastery 157
Dead_Sea 157
Golden_Gate_Bridge 155
Mount_Arapiles 154
Matka_Canyon 153
Burrator 147
Feroz_Shah_Kotla 146
Kazan 146
Kasteel_Amerongen 144
In [0]:
validation_freqs.to_csv(os.path.join(stats_dir, "validation_freqs.csv"), index=True)

Test Data Frequencies and Count :

In [15]:
test_freqs = index_test_csv.get_freqs()
display(test_freqs)
images_count
Masada 158
Hayravank_monastery 157
Dead_Sea 157
Golden_Gate_Bridge 156
Mount_Arapiles 154
Matka_Canyon 152
Feroz_Shah_Kotla 147
Kazan 147
Burrator 146
Kasteel_Amerongen 144
In [0]:
test_freqs.to_csv(os.path.join(stats_dir, "test_freqs.csv"), index=True)

Exploratory Visualization

The horizontal bar plots in show the frequency distribution of classes in training, validation and testing sets. The $y$-axis shows the landmark categories and the $x$-axis represents the frequencies of landmark categories. From the plot we can see that the datasets are well balanced and the image samples are almost uniformly distributed between classes:

In [0]:
plt.figure(figsize=(8, 14))

plt.subplot(311)
index_train_ax = index_train_csv.get_plot()
plt.subplot(312, sharex=index_train_ax)
index_validation_ax = index_validation_csv.get_plot()
plt.subplot(313, sharex=index_train_ax)
index_test_ax = index_test_csv.get_plot()

index_train_ax.xaxis.set_tick_params(which='both', labelleft=True) # Get ticklabels back on shared axis
index_validation_ax.xaxis.set_tick_params(which='both', labelleft=True) # Get ticklabels back on shared axis

plt.savefig(path.join(figures_dir, "dataset_hbar_plot.pdf"), bbox_inches = 'tight')
plt.show()

Algorithms and Techniques

Convolutional neural networks (CNNs) are better suited to image classification tasks than multi-layer perceptrons (MLPs), which only use fully connected layers. MLPs use lots of parameters and the computational complexity of the network can become very large. Another disadvantage is that they discard 2D information of the pixels as they flatten the input image to a 1D vector.

CNNs takes advantage of the spatial proximity of a group of pixels by using sparsely connected layers and accepting a 2D matrix as an input. Pixels close to each other are relevant to the extraction of patterns in the image. Each group of pixels close to each other shares a common group of weights (parameters), the idea being that different regions within the image can share the same kind of information.

The 2D convolution operation is simple: We start with a kernel (or filter), which is simply a small matrix of weights and is the same size as the convolutional window. This kernel slides over the 2D input matrix of image pixels (nodes), performing an elementwise multiplication with the part of the image it is currently on, and then summing up the results into a single output node. a collection of such output nodes is called a convolutional layer.

To define a CNN, the following hyperparameters can be tuned:

  • kernel size: Represents the size of the convolutional window.
  • Number of filters: Each filter detects a single pattern, so to detect more patterns we need to define more filters.
  • Stride: An integer specifies the steps that the sliding convolutional window moves.
  • Padding: Padding the input such that the output has the same length as the original input.

I have chosen the transfer learning technique to solve the given problem. In transfer learning, we build our model to take advantage of some pre-trained CNN model architectures that take hours for supercomputers to train. The fact that a convolutional neural network can learn different features at different layers and that a pre-trained model trained on a task can be used for a similar task is the basis of this technique.

In this problem, I will use VGG16 and Xception pre-trained CNN models from Keras Applications that were trained on the ImageNet dataset without their top layers (fully connected layers) as feature extractors. I will configure my models to apply global average pooling to the output of their last convolutional blocks, so the output model will be a 2D tensor. Then, I will pass my training dataset through each network and save their output bottleneck features. Finally, I will create a top-model for each network that is composed of fully connected layers that take such bottleneck features and output a vector corresponding to our landmark classes (here 10 classes). Transfer learning speeds up training and improves performance significantly.

I will compare my reused pre-trained models to a benchmark traditional convolutional neural network that I'll implement and train from scratch.

Benchmark

Our benchmark convolutional neural network is a simple stack of 3 convolution layers with ReLU activation followed by max-pooling, global average pooling, and dense layers . For the final output dense_ layer, I used the SoftMax activation function to get a vector of probabilities in the $[0,1]$ range. This is very similar to the architectures that Yann LeCun advocated in the 1990s for image classification Handwritten Zip Code Recognition with Multilayer Networks (except for ReLU). I have also added a dropout layer to prevent overfitting.

Benchmark Data Processing

In [16]:
class BenchmarkDataProcessor:
    def __init__(self, input_shape, batch_size, train_dir, vaidation_dir, test_dir):
        self.train_datagen = ImageDataGenerator(
                        rescale = 1.0/255, #rescale pixel values from [0,255] to [0,1]
                        rotation_range=40,
                        width_shift_range=0.2,
                        height_shift_range=0.2,
                        shear_range=0.2,
                        zoom_range=0.2,
                        horizontal_flip=True,
                        fill_mode='nearest')
        self.test_datagen = ImageDataGenerator(rescale=1.0/255)

        self.input_shape = input_shape
        self.batch_size = batch_size

        self.init_train_generator(train_dir)
        self.init_validation_generator(validation_dir)
        self.init_test_generator(test_dir)

    def __init_generator(self, datagen, images_dir):
        return datagen.flow_from_directory(
                        directory=images_dir ,  # this is the target directory
                        target_size=self.input_shape[:2],  # all images will be resized to input shape 224x224
                        color_mode="rgb",
                        batch_size=self.batch_size,
                        class_mode='categorical',
                        shuffle=False)  
    def init_train_generator(self, train_dir):
        self.train_generator = self.__init_generator(self.train_datagen, train_dir)

    def init_validation_generator(self, validation_dir):
        self.validation_generator = self.__init_generator(self.test_datagen, validation_dir)
    def init_test_generator(self, test_dir):
        self.test_generator = self.__init_generator(self.test_datagen, test_dir)    
In [17]:
input_shape=(224, 224, 3)
batch_size = 16
In [0]:
benchmark_data_processor = BenchmarkDataProcessor(
    input_shape, 
    batch_size, 
    train_dir, 
    validation_dir,
    test_dir)
Found 7082 images belonging to 10 classes.
Found 1516 images belonging to 10 classes.
Found 1518 images belonging to 10 classes.

Example data augmentation :

In [0]:
img = load_img(path.join(train_dir, "Kazan/3e222ad7d1469deb.jpg"))
img_array = img_to_array(img)
In [0]:
plt.imshow(img_array/255)
plt.title("Original Image")
plt.savefig(path.join(figures_dir, "augmented_image_original.pdf"), bbox_inches = 'tight')
plt.show()

#----------------------------

columns = 4
rows = 5
fig = plt.figure(figsize=(20,20))

for i, batch in enumerate(benchmark_data_processor.train_datagen.flow(img_array.reshape((1,) + img_array.shape), batch_size=16)):
    if i > rows * columns - 1:
      break
    fig.add_subplot(rows, columns, i+1)
    plt.imshow(batch[0])

plt.savefig(path.join(figures_dir, "augmented_image_transformations.pdf"), bbox_inches = 'tight')
plt.show()

Benchmark Model Architecture

In [10]:
# My benchmark model - a simple CNN
benchmark_model = Sequential()

benchmark_model.add(Conv2D(32, (3, 3), padding='same', activation='relu', input_shape=input_shape))
benchmark_model.add(MaxPooling2D(pool_size=(2, 2)))
benchmark_model.add(Conv2D(32, (3, 3), padding='same', activation='relu'))
benchmark_model.add(MaxPooling2D(pool_size=(2, 2)))
benchmark_model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
benchmark_model.add(MaxPooling2D(pool_size=(2, 2)))
benchmark_model.add(GlobalAveragePooling2D())
#benchmark_model.add(Flatten())
benchmark_model.add(Dense(64, activation='relu'))
benchmark_model.add(Dropout(0.5))
benchmark_model.add(Dense(10, activation='softmax'))


benchmark_model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_4 (Conv2D)            (None, 224, 224, 32)      896       
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 112, 112, 32)      0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 112, 112, 32)      9248      
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 56, 56, 32)        0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 56, 56, 64)        18496     
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 28, 28, 64)        0         
_________________________________________________________________
global_average_pooling2d_2 ( (None, 64)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 64)                4160      
_________________________________________________________________
dropout_2 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_4 (Dense)              (None, 10)                650       
=================================================================
Total params: 33,450
Trainable params: 33,450
Non-trainable params: 0
_________________________________________________________________
In [ ]:
benchmark_model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])
In [9]:
# Save Model Architecture to file
plot_model(benchmark_model,
           to_file=path.join(figures_dir, "benchmark_model_architecture.pdf"),
           show_shapes=True,
           show_layer_names=False
          )

Benchmark Model Training

In [12]:
benchmark_weights_file = path.join(models_dir, "benchmark-model_weights.hdf5")
benchmark_history_file = path.join(models_dir, "benchmark-model_history.pickle")
In [0]:
#Checkpointer to save model best weights
benchmark_checkpointer = ModelCheckpoint(filepath = benchmark_weights_file,
                                         monitor='val_acc',
                                         verbose=1,
                                         save_best_only=True)
In [0]:
#Early Stopping
early_stopping = EarlyStopping(monitor='val_acc',
                               verbose=1,
                               patience=10)
In [0]:
epochs = 50 
steps_per_epoch = benchmark_data_processor.train_generator.samples // benchmark_data_processor.batch_size 
validation_steps = benchmark_data_processor.validation_generator.samples // benchmark_data_processor.batch_size
In [0]:
#epochs = 5
#steps_per_epoch = 15
#validation_steps = 4
In [0]:
benchmark_model_history = benchmark_model.fit_generator(
        benchmark_data_processor.train_generator,
        steps_per_epoch=steps_per_epoch,
        epochs=epochs,
        callbacks = [benchmark_checkpointer, early_stopping],
        validation_data=benchmark_data_processor.validation_generator,
        validation_steps=validation_steps,
        verbose=1)
Epoch 1/50
442/442 [==============================] - 49s 111ms/step - loss: 2.3055 - acc: 0.1015 - val_loss: 2.2559 - val_acc: 0.2207

Epoch 00001: val_acc improved from -inf to 0.22074, saving model to ../models/benchmark-model_weights.hdf5
Epoch 2/50
442/442 [==============================] - 56s 127ms/step - loss: 2.2095 - acc: 0.1903 - val_loss: 2.0548 - val_acc: 0.3040

Epoch 00002: val_acc improved from 0.22074 to 0.30400, saving model to ../models/benchmark-model_weights.hdf5
Epoch 3/50
442/442 [==============================] - 55s 124ms/step - loss: 2.0988 - acc: 0.2503 - val_loss: 1.9875 - val_acc: 0.2933

Epoch 00003: val_acc did not improve from 0.30400
Epoch 4/50
442/442 [==============================] - 51s 115ms/step - loss: 2.0170 - acc: 0.2895 - val_loss: 2.0772 - val_acc: 0.2447

Epoch 00004: val_acc did not improve from 0.30400
Epoch 5/50
442/442 [==============================] - 49s 111ms/step - loss: 1.9388 - acc: 0.3175 - val_loss: 1.8064 - val_acc: 0.4060

Epoch 00005: val_acc improved from 0.30400 to 0.40600, saving model to ../models/benchmark-model_weights.hdf5
Epoch 6/50
442/442 [==============================] - 50s 112ms/step - loss: 1.9007 - acc: 0.3426 - val_loss: 1.7332 - val_acc: 0.4360

Epoch 00006: val_acc improved from 0.40600 to 0.43600, saving model to ../models/benchmark-model_weights.hdf5
Epoch 7/50
442/442 [==============================] - 49s 110ms/step - loss: 1.8590 - acc: 0.3477 - val_loss: 1.7936 - val_acc: 0.3820

Epoch 00007: val_acc did not improve from 0.43600
Epoch 8/50
442/442 [==============================] - 49s 110ms/step - loss: 1.8224 - acc: 0.3657 - val_loss: 1.7011 - val_acc: 0.4113

Epoch 00008: val_acc did not improve from 0.43600
Epoch 9/50
442/442 [==============================] - 48s 109ms/step - loss: 1.7863 - acc: 0.3923 - val_loss: 1.6496 - val_acc: 0.4320

Epoch 00009: val_acc did not improve from 0.43600
Epoch 10/50
442/442 [==============================] - 48s 109ms/step - loss: 1.7482 - acc: 0.4045 - val_loss: 1.9517 - val_acc: 0.3227

Epoch 00010: val_acc did not improve from 0.43600
Epoch 11/50
442/442 [==============================] - 48s 109ms/step - loss: 1.7443 - acc: 0.3983 - val_loss: 1.8431 - val_acc: 0.3740

Epoch 00011: val_acc did not improve from 0.43600
Epoch 12/50
442/442 [==============================] - 48s 109ms/step - loss: 1.6896 - acc: 0.4273 - val_loss: 1.5942 - val_acc: 0.4787

Epoch 00012: val_acc improved from 0.43600 to 0.47867, saving model to ../models/benchmark-model_weights.hdf5
Epoch 13/50
442/442 [==============================] - 48s 108ms/step - loss: 1.6952 - acc: 0.4150 - val_loss: 1.5623 - val_acc: 0.4687

Epoch 00013: val_acc did not improve from 0.47867
Epoch 14/50
442/442 [==============================] - 48s 110ms/step - loss: 1.6671 - acc: 0.4369 - val_loss: 1.5351 - val_acc: 0.4707

Epoch 00014: val_acc did not improve from 0.47867
Epoch 15/50
442/442 [==============================] - 49s 110ms/step - loss: 1.6543 - acc: 0.4316 - val_loss: 1.5297 - val_acc: 0.4747

Epoch 00015: val_acc did not improve from 0.47867
Epoch 16/50
442/442 [==============================] - 48s 109ms/step - loss: 1.6373 - acc: 0.4496 - val_loss: 1.4702 - val_acc: 0.5173

Epoch 00016: val_acc improved from 0.47867 to 0.51733, saving model to ../models/benchmark-model_weights.hdf5
Epoch 17/50
442/442 [==============================] - 48s 109ms/step - loss: 1.6023 - acc: 0.4523 - val_loss: 1.4707 - val_acc: 0.5033

Epoch 00017: val_acc did not improve from 0.51733
Epoch 18/50
442/442 [==============================] - 49s 112ms/step - loss: 1.5865 - acc: 0.4571 - val_loss: 1.4492 - val_acc: 0.5187

Epoch 00018: val_acc improved from 0.51733 to 0.51867, saving model to ../models/benchmark-model_weights.hdf5
Epoch 19/50
442/442 [==============================] - 49s 110ms/step - loss: 1.5659 - acc: 0.4725 - val_loss: 1.4891 - val_acc: 0.4900

Epoch 00019: val_acc did not improve from 0.51867
Epoch 20/50
442/442 [==============================] - 48s 109ms/step - loss: 1.5523 - acc: 0.4727 - val_loss: 1.4055 - val_acc: 0.5300

Epoch 00020: val_acc improved from 0.51867 to 0.53000, saving model to ../models/benchmark-model_weights.hdf5
Epoch 21/50
442/442 [==============================] - 48s 109ms/step - loss: 1.5292 - acc: 0.4788 - val_loss: 1.3411 - val_acc: 0.5447

Epoch 00021: val_acc improved from 0.53000 to 0.54467, saving model to ../models/benchmark-model_weights.hdf5
Epoch 22/50
442/442 [==============================] - 48s 109ms/step - loss: 1.5130 - acc: 0.4861 - val_loss: 1.3737 - val_acc: 0.5280

Epoch 00022: val_acc did not improve from 0.54467
Epoch 23/50
442/442 [==============================] - 48s 109ms/step - loss: 1.4761 - acc: 0.4985 - val_loss: 1.4344 - val_acc: 0.5033

Epoch 00023: val_acc did not improve from 0.54467
Epoch 24/50
442/442 [==============================] - 48s 109ms/step - loss: 1.4643 - acc: 0.5051 - val_loss: 1.3136 - val_acc: 0.5580

Epoch 00024: val_acc improved from 0.54467 to 0.55800, saving model to ../models/benchmark-model_weights.hdf5
Epoch 25/50
442/442 [==============================] - 49s 110ms/step - loss: 1.4559 - acc: 0.5051 - val_loss: 1.4510 - val_acc: 0.5093

Epoch 00025: val_acc did not improve from 0.55800
Epoch 26/50
442/442 [==============================] - 48s 109ms/step - loss: 1.4240 - acc: 0.5257 - val_loss: 1.3582 - val_acc: 0.5260

Epoch 00026: val_acc did not improve from 0.55800
Epoch 27/50
442/442 [==============================] - 48s 109ms/step - loss: 1.4310 - acc: 0.5156 - val_loss: 1.2784 - val_acc: 0.5827

Epoch 00027: val_acc improved from 0.55800 to 0.58267, saving model to ../models/benchmark-model_weights.hdf5
Epoch 28/50
442/442 [==============================] - 48s 109ms/step - loss: 1.4224 - acc: 0.5211 - val_loss: 1.2739 - val_acc: 0.5767

Epoch 00028: val_acc did not improve from 0.58267
Epoch 29/50
442/442 [==============================] - 48s 110ms/step - loss: 1.3860 - acc: 0.5445 - val_loss: 1.2414 - val_acc: 0.5747

Epoch 00029: val_acc did not improve from 0.58267
Epoch 30/50
442/442 [==============================] - 49s 110ms/step - loss: 1.3947 - acc: 0.5338 - val_loss: 1.3700 - val_acc: 0.5373

Epoch 00030: val_acc did not improve from 0.58267
Epoch 31/50
442/442 [==============================] - 48s 110ms/step - loss: 1.3693 - acc: 0.5430 - val_loss: 1.2721 - val_acc: 0.5620

Epoch 00031: val_acc did not improve from 0.58267
Epoch 32/50
442/442 [==============================] - 48s 109ms/step - loss: 1.3563 - acc: 0.5518 - val_loss: 1.3003 - val_acc: 0.5467

Epoch 00032: val_acc did not improve from 0.58267
Epoch 33/50
442/442 [==============================] - 48s 109ms/step - loss: 1.3364 - acc: 0.5538 - val_loss: 1.3020 - val_acc: 0.5673

Epoch 00033: val_acc did not improve from 0.58267
Epoch 34/50
442/442 [==============================] - 48s 110ms/step - loss: 1.3523 - acc: 0.5484 - val_loss: 1.3013 - val_acc: 0.5653

Epoch 00034: val_acc did not improve from 0.58267
Epoch 35/50
442/442 [==============================] - 49s 110ms/step - loss: 1.3361 - acc: 0.5605 - val_loss: 1.1603 - val_acc: 0.6133

Epoch 00035: val_acc improved from 0.58267 to 0.61333, saving model to ../models/benchmark-model_weights.hdf5
Epoch 36/50
442/442 [==============================] - 49s 110ms/step - loss: 1.2873 - acc: 0.5668 - val_loss: 1.3825 - val_acc: 0.5653

Epoch 00036: val_acc did not improve from 0.61333
Epoch 37/50
442/442 [==============================] - 49s 111ms/step - loss: 1.3112 - acc: 0.5605 - val_loss: 1.1409 - val_acc: 0.6253

Epoch 00037: val_acc improved from 0.61333 to 0.62533, saving model to ../models/benchmark-model_weights.hdf5
Epoch 38/50
442/442 [==============================] - 50s 112ms/step - loss: 1.2895 - acc: 0.5669 - val_loss: 1.2113 - val_acc: 0.5953

Epoch 00038: val_acc did not improve from 0.62533
Epoch 39/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2858 - acc: 0.5731 - val_loss: 1.2004 - val_acc: 0.5840

Epoch 00039: val_acc did not improve from 0.62533
Epoch 40/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2734 - acc: 0.5745 - val_loss: 1.2686 - val_acc: 0.5720

Epoch 00040: val_acc did not improve from 0.62533
Epoch 41/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2582 - acc: 0.5921 - val_loss: 1.3580 - val_acc: 0.5360

Epoch 00041: val_acc did not improve from 0.62533
Epoch 42/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2409 - acc: 0.5877 - val_loss: 1.4456 - val_acc: 0.5180

Epoch 00042: val_acc did not improve from 0.62533
Epoch 43/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2500 - acc: 0.5843 - val_loss: 1.1137 - val_acc: 0.6367

Epoch 00043: val_acc improved from 0.62533 to 0.63667, saving model to ../models/benchmark-model_weights.hdf5
Epoch 44/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2474 - acc: 0.5894 - val_loss: 1.1284 - val_acc: 0.6127

Epoch 00044: val_acc did not improve from 0.63667
Epoch 45/50
442/442 [==============================] - 49s 111ms/step - loss: 1.2165 - acc: 0.5951 - val_loss: 1.1701 - val_acc: 0.6060

Epoch 00045: val_acc did not improve from 0.63667
Epoch 46/50
442/442 [==============================] - 48s 109ms/step - loss: 1.2336 - acc: 0.5876 - val_loss: 1.1086 - val_acc: 0.6313

Epoch 00046: val_acc did not improve from 0.63667
Epoch 47/50
442/442 [==============================] - 49s 110ms/step - loss: 1.2081 - acc: 0.6083 - val_loss: 1.8022 - val_acc: 0.4700

Epoch 00047: val_acc did not improve from 0.63667
Epoch 48/50
442/442 [==============================] - 49s 110ms/step - loss: 1.2205 - acc: 0.6029 - val_loss: 1.0908 - val_acc: 0.6313

Epoch 00048: val_acc did not improve from 0.63667
Epoch 49/50
442/442 [==============================] - 49s 110ms/step - loss: 1.1987 - acc: 0.6048 - val_loss: 1.1599 - val_acc: 0.6200

Epoch 00049: val_acc did not improve from 0.63667
Epoch 50/50
442/442 [==============================] - 49s 111ms/step - loss: 1.1859 - acc: 0.6045 - val_loss: 1.1850 - val_acc: 0.6000

Epoch 00050: val_acc did not improve from 0.63667
In [0]:
# Save Model History
with open(benchmark_history_file, 'wb') as pickle_file:
    pickle.dump(benchmark_model_history, pickle_file)
In [14]:
# Load Model History
with open(benchmark_history_file, 'rb') as pickle_file:
    benchmark_model_history = pickle.load(pickle_file)
In [0]:
# Loading Best Weights
benchmark_model.load_weights(benchmark_weights_file)

Benchmark Model Visualization (Learning Curves) :

In [15]:
def plot_learning_curves(model_history, model_name, plot_filename):
  fig, ax = plt.subplots(2, 1)
  fig.set_size_inches(8, 12)
  #fig.suptitle(model_name + ' Performance Metrics')
  ax[0].plot(model_history.history['acc'])
  ax[0].plot(model_history.history['val_acc'])
  ax[0].set_title(model_name + ' Performance Metrics\n' + 'Accuracy')
  ax[0].legend(['Training', 'Validation'], loc='upper left')
  ax[0].set_xlabel('Epoch')
  ax[0].set_ylabel('Accuracy')
  ax[0].grid(True)
  ax[1].plot(model_history.history['loss'])
  ax[1].plot(model_history.history['val_loss'])
  ax[1].set_title('Loss')
  ax[1].legend(['Training', 'Validation'], loc='upper right')
  ax[1].set_xlabel('Epoch')
  ax[1].set_ylabel('Loss')
  ax[1].grid(True)
  plt.savefig(plot_filename, bbox_inches = 'tight')
  plt.show()
In [16]:
plot_learning_curves(benchmark_model_history, 'Benchmark Model', path.join(figures_dir, "benchmark_model_metrics.pdf"))

Benchmark Model Testing :

Accuracy :

In [0]:
[benchmark_test_loss, benchmark_test_accuracy] = benchmark_model.evaluate_generator(benchmark_data_processor.test_generator, benchmark_data_processor.test_generator.samples, workers=2, use_multiprocessing=True, verbose=1)
print(benchmark_model.metrics_names)
print([benchmark_test_loss, benchmark_test_accuracy])
1518/1518 [==============================] - 152s 100ms/step
['loss', 'acc']
[1.1270842477193972, 0.6429219225909746]

GAP :

In [0]:
def get_GAP(model, test_generator):
  test_generator.reset()
  samples_count = test_generator.samples
  images = np.empty((0,) + input_shape)
  max_batch_index = len(test_generator)
  i = 0
  for batch in test_generator:
      images = np.append(images, batch[0], axis=0) #image data
  #   print(batch[1][i]) #image labels
      i += 1
      if i > max_batch_index - 1:
          break  

  probabilities = model.predict(images)
  predicted_classes = model.predict_classes(images)
  confidence_scores = probabilities[(np.arange(samples_count), predicted_classes)]
  true_labels = test_generator.labels
  return GAP_vector(predicted_classes, confidence_scores, true_labels)
In [0]:
benchmark_test_GAP = get_GAP(benchmark_model, benchmark_data_processor.test_generator)
print(benchmark_test_GAP)
0.558556285543901

Saving results :

In [0]:
stats_csv.add_stats("Benchmark Model", benchmark_test_loss, benchmark_test_accuracy, benchmark_test_GAP)

III. Methodology

Data Preprocessing

I defined a DataProcessor class that wraps Keras ImageDataGenerator() and has attributes like train_generator, validation_generator, and test_generator that act as directory iterators. They return batches of augmented images. This class also handles image augmentation and pixel value re-scaling to the range $[0,1]$

The following code is the implementation of DataProcessor class :

In [19]:
class DataProcessor:
    def __init__(self, input_shape, batch_size, train_dir, validation_dir, test_dir):
        self.train_datagen = ImageDataGenerator(rescale = 1.0/255) #rescale pixel values from [0,255] to [0,1]
        self.test_datagen = ImageDataGenerator(rescale=1.0/255)

        self.input_shape = input_shape
        self.batch_size = batch_size

        self.init_train_generator(train_dir)
        self.init_validation_generator(validation_dir)
        self.init_test_generator(test_dir)

    def __init_generator(self, datagen, images_dir):
        return datagen.flow_from_directory(
                        directory=images_dir ,  # this is the target directory
                        target_size=self.input_shape[:2],  # all images will be resized to input shape 224x224
                        batch_size=self.batch_size,
                        class_mode=None,
                        shuffle=False)  
    def init_train_generator(self, train_dir):
        self.train_generator = self.__init_generator(self.train_datagen, train_dir)

    def init_validation_generator(self, validation_dir):
        self.validation_generator = self.__init_generator(self.test_datagen, validation_dir)
    def init_test_generator(self, test_dir):
        self.test_generator = self.__init_generator(self.test_datagen, test_dir)    
In [18]:
input_shape=(224, 224, 3)
batch_size = 1
In [20]:
data_processor = DataProcessor(
    input_shape, 
    batch_size, 
    train_dir, 
    validation_dir,
    test_dir)
Found 7082 images belonging to 10 classes.
Found 1516 images belonging to 10 classes.
Found 1518 images belonging to 10 classes.

Implementation

I also defined a PretrainedModel class that wraps a pre-trained transfer learning model and our defined top-model. It has methods for :

  • Calculation of bottleneck features.
  • Saving and loading bottleneck features to .npz files.
  • Top-model creation, compiling and training.
  • Saving and loading top-model history and best weights.
  • Calculation of test accuracy and GAP scores (\nameref{sec:metrics}).
  • Plotting learning curves.

The following code is the implementation of PretrainedModel class :

In [27]:
class PretrainedModel:
    def __init__(self, model_name, pretrained_model, data_processor, save_path ):
        self.model_name = model_name
        self.pretrained_model = pretrained_model
        self.data_processor = data_processor
        self.save_path = save_path

    def get_file_path(self, file_name):
      return path.join(self.save_path, self.model_name + '_' + file_name)
        
    def predict_bottleneck_features(self):
        max_queue_size = 10    #defult is 10
        workers = 2    # Default is 1
        self.bottleneck_features_train = self.pretrained_model.predict_generator(
            self.data_processor.train_generator,
            steps=self.data_processor.train_generator.samples // self.data_processor.batch_size,
            max_queue_size=max_queue_size,
            workers=workers,
            use_multiprocessing=True, #default is False
            verbose=1)     
        self.bottleneck_features_validation = self.pretrained_model.predict_generator(
            self.data_processor.validation_generator, 
            steps=self.data_processor.validation_generator.samples // self.data_processor.batch_size,
            max_queue_size=max_queue_size,
            workers=workers,
            use_multiprocessing=True,
            verbose=1)
        self.bottleneck_features_test = self.pretrained_model.predict_generator(
            self.data_processor.test_generator, 
            steps=self.data_processor.test_generator.samples // self.data_processor.batch_size,
            max_queue_size=max_queue_size,
            workers=workers,
            use_multiprocessing=True,
            verbose=1)
     
        self.save_bottleneck_features()

    def save_bottleneck_features(self):
        np.save(open(self.get_file_path('bottleneck_features_train.npz'), 'wb'),
                self.bottleneck_features_train)
        np.save(open(self.get_file_path('bottleneck_features_validation.npz'), 'wb'),
                self.bottleneck_features_validation)
        np.save(open(self.get_file_path('bottleneck_features_test.npz'), 'wb'),
                self.bottleneck_features_test)

    def load_bottleneck_features(self):
        self.bottleneck_features_train = np.load(open(self.get_file_path('bottleneck_features_train.npz'), 'rb'))
        self.bottleneck_features_validation = np.load(open(self.get_file_path('bottleneck_features_validation.npz'), 'rb'))
        self.bottleneck_features_test = np.load(open(self.get_file_path('bottleneck_features_test.npz'), 'rb'))

    def create_top_model(self, optimizer):
        '''
        defines a fully connected top model that has three Dense layers with input size equals to bottleneck
        features. it takes bottleneck features as input and outputs a vector of 10 classes corresponding to
        our 10 landmark categories. I also used a Dropout layer to reduce overfitting .
        I used ReLU activation for the first and second Dense layers and Softmax activation for the last 
        layer to create a vector of probabilities with values between 0 and 1.
        '''
        self.top_model = Sequential()
        self.top_model.add(Dense(256, activation='relu', input_shape=self.bottleneck_features_train.shape[1:])) #[1:]
        self.top_model.add(Dense(128, activation='relu'))
        self.top_model.add(Dropout(0.3))
        self.top_model.add(Dense(10, activation='softmax'))

        self.top_model.compile(optimizer=optimizer,
                               loss='categorical_crossentropy',
                               metrics=['accuracy'])
        self.top_model.summary()
        
    def save_top_model_graph(self, figures_path): # Save Top Model Architecture to file
        plot_model(self.top_model,
                   to_file=path.join(figures_path, self.model_name +"_top-model_architecture.pdf"),
                   show_shapes=True,
                   show_layer_names=False
          )

    def train_top_model(self, epochs, batch_size):
        early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=100)
        
        checkpointer =  ModelCheckpoint(
            filepath=self.get_file_path('top-model_weights.hdf5'),
            monitor='val_acc',
            verbose=0,
            save_best_only=True)
        
        train_labels = to_categorical(self.data_processor.train_generator.classes)
        validation_labels = to_categorical(self.data_processor.validation_generator.classes)
        
        self.history = self.top_model.fit(self.bottleneck_features_train,
                                 train_labels,
                                 epochs=epochs,
                                 batch_size=batch_size,
                                 validation_data=(self.bottleneck_features_validation, validation_labels),
                                 callbacks=[checkpointer, early_stopping],
                                 verbose=1)

    def save_top_model_history(self):
      with open(self.get_file_path('top-model_history.pickle'), 'wb') as pickle_file:
        pickle.dump(self.history, pickle_file)

    def load_top_model_history(self):
      with open(self.get_file_path('top-model_history.pickle'), 'rb') as pickle_file:
        self.history = pickle.load(pickle_file)

    def load_top_model_weights(self):
        self.top_model.load_weights(self.get_file_path('top-model_weights.hdf5'))
        
    def test_top_model(self):
        self.load_top_model_weights()
        test_labels = to_categorical(self.data_processor.test_generator.classes)
        stats = self.top_model.evaluate(self.bottleneck_features_test, test_labels, workers=2, use_multiprocessing=True, verbose=1)
        print(stats)
        return stats

    def get_GAP(self):
      samples_count = self.data_processor.test_generator.samples
      probabilities = self.top_model.predict(self.bottleneck_features_test)
      predicted_classes = self.top_model.predict_classes(self.bottleneck_features_test)
      confidence_scores = probabilities[(np.arange(samples_count), predicted_classes)]
      true_labels = self.data_processor.test_generator.labels
      GAP = GAP_vector(predicted_classes, confidence_scores, true_labels)
      print(GAP)
      return GAP

    def plot_learning_curves(self, save_path):
      fig, ax = plt.subplots(1, 2)
      fig.set_size_inches(16,4)
      fig.suptitle(self.model_name + ' Performance Metrics')
      ax[0].plot(self.history.history['acc'])
      ax[0].plot(self.history.history['val_acc'])
      ax[0].set_title('Accuracy')
      ax[0].legend(['Training', 'Validation'], loc='upper left')
      ax[0].set_xlabel('Epoch')
      ax[0].set_ylabel('Accuracy')
      ax[0].grid(True)
      ax[1].plot(self.history.history['loss'])
      ax[1].plot(self.history.history['val_loss'])
      ax[1].set_title('Loss')
      ax[1].legend(['Training', 'Validation'], loc='upper right')
      ax[1].set_xlabel('Epoch')
      ax[1].set_ylabel('Loss')
      ax[1].grid(True)
      plt.savefig(path.join(save_path, self.model_name +"_metrics.pdf"), bbox_inches = 'tight')
      plt.show()

Pretrained VGG16 Model :

Model Creation :

Create an instance of PretrainedModel class, passing VGG16 models instance without its top layers (include_top=False) and with average pooling mode for feature extraction (pooling='avg') and an object of DataProcessor class :

In [28]:
VGG16_model = PretrainedModel("VGG16_model",
                             applications.VGG16(include_top=False, weights='imagenet',pooling='avg'),
                             data_processor,
                             models_dir)
Model Training :
In [0]:
VGG16_model.predict_bottleneck_features()
7082/7082 [==============================] - 85s 12ms/step
1516/1516 [==============================] - 19s 12ms/step
1518/1518 [==============================] - 18s 12ms/step
In [29]:
VGG16_model.load_bottleneck_features()

Create and compile the top-model passing your optimizer of choice to create_top_model() function. In my initial solution I used stochastic gradient descent (SGD) optimizer

In [30]:
VGG16_model.create_top_model(optimizers.SGD(lr=0.01, clipnorm=1.,momentum=0.7))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_8 (Dense)              (None, 256)               131328    
_________________________________________________________________
dense_9 (Dense)              (None, 128)               32896     
_________________________________________________________________
dropout_4 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_10 (Dense)             (None, 10)                1290      
=================================================================
Total params: 165,514
Trainable params: 165,514
Non-trainable params: 0
_________________________________________________________________
In [35]:
# Save top-model architecture to file
VGG16_model.save_top_model_graph(figures_dir)

Fit (train) the model using the bottleneck features for 1000 epochs(we can choose a large number of epochs as training the dense layers is very fast, unlike our benchmark model that has convolutional layers) :

In [43]:
VGG16_model.train_top_model(epochs=1000, batch_size=4096)
Epoch 951/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9508 - acc: 0.6810 - val_loss: 1.0399 - val_acc: 0.6583

Epoch 00951: val_acc did not improve from 0.66029
Epoch 952/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9606 - acc: 0.6837 - val_loss: 1.0396 - val_acc: 0.6530

Epoch 00952: val_acc did not improve from 0.66029
Epoch 953/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9607 - acc: 0.6836 - val_loss: 1.0381 - val_acc: 0.6550

Epoch 00953: val_acc did not improve from 0.66029
Epoch 954/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9560 - acc: 0.6877 - val_loss: 1.0378 - val_acc: 0.6596

Epoch 00954: val_acc did not improve from 0.66029
Epoch 955/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9528 - acc: 0.6889 - val_loss: 1.0397 - val_acc: 0.6563

Epoch 00955: val_acc did not improve from 0.66029
Epoch 956/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9585 - acc: 0.6889 - val_loss: 1.0379 - val_acc: 0.6609

Epoch 00956: val_acc improved from 0.66029 to 0.66095, saving model to ../models/VGG16_model_top-model_weights.hdf5
Epoch 957/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9529 - acc: 0.6827 - val_loss: 1.0377 - val_acc: 0.6596

Epoch 00957: val_acc did not improve from 0.66095
Epoch 958/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9523 - acc: 0.6837 - val_loss: 1.0383 - val_acc: 0.6590

Epoch 00958: val_acc did not improve from 0.66095
Epoch 959/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9493 - acc: 0.6844 - val_loss: 1.0374 - val_acc: 0.6557

Epoch 00959: val_acc did not improve from 0.66095
Epoch 960/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9537 - acc: 0.6802 - val_loss: 1.0386 - val_acc: 0.6583

Epoch 00960: val_acc did not improve from 0.66095
Epoch 961/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9576 - acc: 0.6826 - val_loss: 1.0366 - val_acc: 0.6577

Epoch 00961: val_acc did not improve from 0.66095
Epoch 962/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9561 - acc: 0.6871 - val_loss: 1.0373 - val_acc: 0.6603

Epoch 00962: val_acc did not improve from 0.66095
Epoch 963/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9537 - acc: 0.6855 - val_loss: 1.0367 - val_acc: 0.6603

Epoch 00963: val_acc did not improve from 0.66095
Epoch 964/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9456 - acc: 0.6877 - val_loss: 1.0397 - val_acc: 0.6497

Epoch 00964: val_acc did not improve from 0.66095
Epoch 965/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9545 - acc: 0.6875 - val_loss: 1.0378 - val_acc: 0.6603

Epoch 00965: val_acc did not improve from 0.66095
Epoch 966/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9487 - acc: 0.6864 - val_loss: 1.0369 - val_acc: 0.6544

Epoch 00966: val_acc did not improve from 0.66095
Epoch 967/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9487 - acc: 0.6872 - val_loss: 1.0364 - val_acc: 0.6577

Epoch 00967: val_acc did not improve from 0.66095
Epoch 968/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9541 - acc: 0.6834 - val_loss: 1.0347 - val_acc: 0.6577

Epoch 00968: val_acc did not improve from 0.66095
Epoch 969/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9473 - acc: 0.6853 - val_loss: 1.0349 - val_acc: 0.6590

Epoch 00969: val_acc did not improve from 0.66095
Epoch 970/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9428 - acc: 0.6919 - val_loss: 1.0340 - val_acc: 0.6603

Epoch 00970: val_acc did not improve from 0.66095
Epoch 971/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9414 - acc: 0.6894 - val_loss: 1.0340 - val_acc: 0.6577

Epoch 00971: val_acc did not improve from 0.66095
Epoch 972/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9418 - acc: 0.6881 - val_loss: 1.0337 - val_acc: 0.6629

Epoch 00972: val_acc improved from 0.66095 to 0.66293, saving model to ../models/VGG16_model_top-model_weights.hdf5
Epoch 973/1000
7082/7082 [==============================] - 0s 5us/step - loss: 0.9427 - acc: 0.6896 - val_loss: 1.0333 - val_acc: 0.6596

Epoch 00973: val_acc did not improve from 0.66293
Epoch 974/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9513 - acc: 0.6805 - val_loss: 1.0358 - val_acc: 0.6557

Epoch 00974: val_acc did not improve from 0.66293
Epoch 975/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9404 - acc: 0.6882 - val_loss: 1.0325 - val_acc: 0.6603

Epoch 00975: val_acc did not improve from 0.66293
Epoch 976/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9402 - acc: 0.6903 - val_loss: 1.0331 - val_acc: 0.6596

Epoch 00976: val_acc did not improve from 0.66293
Epoch 977/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9483 - acc: 0.6855 - val_loss: 1.0329 - val_acc: 0.6609

Epoch 00977: val_acc did not improve from 0.66293
Epoch 978/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9490 - acc: 0.6901 - val_loss: 1.0324 - val_acc: 0.6557

Epoch 00978: val_acc did not improve from 0.66293
Epoch 979/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9498 - acc: 0.6881 - val_loss: 1.0327 - val_acc: 0.6623

Epoch 00979: val_acc did not improve from 0.66293
Epoch 980/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9435 - acc: 0.6874 - val_loss: 1.0367 - val_acc: 0.6550

Epoch 00980: val_acc did not improve from 0.66293
Epoch 981/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9417 - acc: 0.6871 - val_loss: 1.0318 - val_acc: 0.6616

Epoch 00981: val_acc did not improve from 0.66293
Epoch 982/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9582 - acc: 0.6798 - val_loss: 1.0336 - val_acc: 0.6583

Epoch 00982: val_acc did not improve from 0.66293
Epoch 983/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9489 - acc: 0.6894 - val_loss: 1.0329 - val_acc: 0.6629

Epoch 00983: val_acc did not improve from 0.66293
Epoch 984/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9429 - acc: 0.6881 - val_loss: 1.0339 - val_acc: 0.6537

Epoch 00984: val_acc did not improve from 0.66293
Epoch 985/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9486 - acc: 0.6871 - val_loss: 1.0314 - val_acc: 0.6609

Epoch 00985: val_acc did not improve from 0.66293
Epoch 986/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9493 - acc: 0.6889 - val_loss: 1.0310 - val_acc: 0.6596

Epoch 00986: val_acc did not improve from 0.66293
Epoch 987/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9477 - acc: 0.6868 - val_loss: 1.0309 - val_acc: 0.6596

Epoch 00987: val_acc did not improve from 0.66293
Epoch 988/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9355 - acc: 0.6970 - val_loss: 1.0299 - val_acc: 0.6603

Epoch 00988: val_acc did not improve from 0.66293
Epoch 989/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9512 - acc: 0.6857 - val_loss: 1.0309 - val_acc: 0.6590

Epoch 00989: val_acc did not improve from 0.66293
Epoch 990/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9445 - acc: 0.6936 - val_loss: 1.0296 - val_acc: 0.6603

Epoch 00990: val_acc did not improve from 0.66293
Epoch 991/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9424 - acc: 0.6840 - val_loss: 1.0294 - val_acc: 0.6596

Epoch 00991: val_acc did not improve from 0.66293
Epoch 992/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9427 - acc: 0.6902 - val_loss: 1.0290 - val_acc: 0.6596

Epoch 00992: val_acc did not improve from 0.66293
Epoch 993/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9401 - acc: 0.6932 - val_loss: 1.0293 - val_acc: 0.6616

Epoch 00993: val_acc did not improve from 0.66293
Epoch 994/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9376 - acc: 0.6884 - val_loss: 1.0290 - val_acc: 0.6590

Epoch 00994: val_acc did not improve from 0.66293
Epoch 995/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9430 - acc: 0.6905 - val_loss: 1.0298 - val_acc: 0.6577

Epoch 00995: val_acc did not improve from 0.66293
Epoch 996/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9474 - acc: 0.6855 - val_loss: 1.0280 - val_acc: 0.6583

Epoch 00996: val_acc did not improve from 0.66293
Epoch 997/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9452 - acc: 0.6836 - val_loss: 1.0287 - val_acc: 0.6609

Epoch 00997: val_acc did not improve from 0.66293
Epoch 998/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9482 - acc: 0.6871 - val_loss: 1.0282 - val_acc: 0.6596

Epoch 00998: val_acc did not improve from 0.66293
Epoch 999/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.9401 - acc: 0.6908 - val_loss: 1.0286 - val_acc: 0.6629

Epoch 00999: val_acc did not improve from 0.66293
Epoch 1000/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.9417 - acc: 0.6891 - val_loss: 1.0290 - val_acc: 0.6563

Epoch 01000: val_acc did not improve from 0.66293
In [ ]:
VGG16_model.save_top_model_history()
In [31]:
VGG16_model.load_top_model_history()
In [0]:
 VGG16_model.load_top_model_weights()
Model Testing :
In [46]:
[test_loss, test_accuracy] = VGG16_model.test_top_model()
1518/1518 [==============================] - 0s 46us/step
[0.9676066144652988, 0.694334650699329]
In [47]:
test_GAP = VGG16_model.get_GAP()
0.6225644520166035
In [0]:
stats_csv.add_stats("VGG16 Model", test_loss, test_accuracy, test_GAP)
Model Visualization :
In [32]:
VGG16_model.plot_learning_curves(figures_dir)

Pretrained Xception Model :

Model Creation :
In [34]:
Xception_model = PretrainedModel("Xception_model",
                             applications.Xception(include_top=False, weights='imagenet',pooling='avg'),
                             data_processor,
                             models_dir)
Model Training :
In [52]:
Xception_model.predict_bottleneck_features()
7082/7082 [==============================] - 117s 17ms/step
1516/1516 [==============================] - 25s 17ms/step
1518/1518 [==============================] - 25s 17ms/step
In [35]:
Xception_model.load_bottleneck_features()
In [36]:
Xception_model.create_top_model(optimizers.SGD(lr=0.01, clipnorm=1.,momentum=0.7))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_11 (Dense)             (None, 256)               524544    
_________________________________________________________________
dense_12 (Dense)             (None, 128)               32896     
_________________________________________________________________
dropout_5 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_13 (Dense)             (None, 10)                1290      
=================================================================
Total params: 558,730
Trainable params: 558,730
Non-trainable params: 0
_________________________________________________________________
In [40]:
# Save top-model architecture to file
Xception_model.save_top_model_graph(figures_dir)
In [57]:
Xception_model.train_top_model(epochs=1000, batch_size=4096)
Epoch 965/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2172 - acc: 0.9360 - val_loss: 0.5723 - val_acc: 0.8285
Epoch 966/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2177 - acc: 0.9389 - val_loss: 0.5722 - val_acc: 0.8278
Epoch 967/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2177 - acc: 0.9376 - val_loss: 0.5726 - val_acc: 0.8259
Epoch 968/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2146 - acc: 0.9391 - val_loss: 0.5731 - val_acc: 0.8252
Epoch 969/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2225 - acc: 0.9350 - val_loss: 0.5731 - val_acc: 0.8272
Epoch 970/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2193 - acc: 0.9360 - val_loss: 0.5731 - val_acc: 0.8265
Epoch 971/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2169 - acc: 0.9397 - val_loss: 0.5732 - val_acc: 0.8252
Epoch 972/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2171 - acc: 0.9387 - val_loss: 0.5734 - val_acc: 0.8245
Epoch 973/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2160 - acc: 0.9393 - val_loss: 0.5733 - val_acc: 0.8245
Epoch 974/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2178 - acc: 0.9367 - val_loss: 0.5731 - val_acc: 0.8259
Epoch 975/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2142 - acc: 0.9384 - val_loss: 0.5726 - val_acc: 0.8259
Epoch 976/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2165 - acc: 0.9398 - val_loss: 0.5728 - val_acc: 0.8259
Epoch 977/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2162 - acc: 0.9389 - val_loss: 0.5730 - val_acc: 0.8265
Epoch 978/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2202 - acc: 0.9355 - val_loss: 0.5732 - val_acc: 0.8265
Epoch 979/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2172 - acc: 0.9376 - val_loss: 0.5729 - val_acc: 0.8285
Epoch 980/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2179 - acc: 0.9376 - val_loss: 0.5726 - val_acc: 0.8265
Epoch 981/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2166 - acc: 0.9362 - val_loss: 0.5728 - val_acc: 0.8259
Epoch 982/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2154 - acc: 0.9386 - val_loss: 0.5723 - val_acc: 0.8252
Epoch 983/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2150 - acc: 0.9394 - val_loss: 0.5722 - val_acc: 0.8259
Epoch 984/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2141 - acc: 0.9372 - val_loss: 0.5724 - val_acc: 0.8252
Epoch 985/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2171 - acc: 0.9396 - val_loss: 0.5727 - val_acc: 0.8245
Epoch 986/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2139 - acc: 0.9372 - val_loss: 0.5729 - val_acc: 0.8272
Epoch 987/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2142 - acc: 0.9366 - val_loss: 0.5726 - val_acc: 0.8259
Epoch 988/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2179 - acc: 0.9386 - val_loss: 0.5726 - val_acc: 0.8272
Epoch 989/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2097 - acc: 0.9414 - val_loss: 0.5725 - val_acc: 0.8278
Epoch 990/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2120 - acc: 0.9380 - val_loss: 0.5725 - val_acc: 0.8245
Epoch 991/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2129 - acc: 0.9406 - val_loss: 0.5726 - val_acc: 0.8252
Epoch 992/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2158 - acc: 0.9383 - val_loss: 0.5728 - val_acc: 0.8252
Epoch 993/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2113 - acc: 0.9389 - val_loss: 0.5727 - val_acc: 0.8245
Epoch 994/1000
7082/7082 [==============================] - 0s 8us/step - loss: 0.2098 - acc: 0.9397 - val_loss: 0.5730 - val_acc: 0.8245
Epoch 995/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2059 - acc: 0.9424 - val_loss: 0.5734 - val_acc: 0.8239
Epoch 996/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2111 - acc: 0.9397 - val_loss: 0.5733 - val_acc: 0.8272
Epoch 997/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2106 - acc: 0.9367 - val_loss: 0.5729 - val_acc: 0.8272
Epoch 998/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2115 - acc: 0.9393 - val_loss: 0.5728 - val_acc: 0.8272
Epoch 999/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.2118 - acc: 0.9420 - val_loss: 0.5731 - val_acc: 0.8272
Epoch 1000/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.2114 - acc: 0.9393 - val_loss: 0.5732 - val_acc: 0.8259
In [ ]:
Xception_model.save_top_model_history()
In [37]:
Xception_model.load_top_model_history()
In [0]:
 Xception_model.load_top_model_weights()
Model Testing :
In [60]:
[test_loss, test_accuracy] = Xception_model.test_top_model()
1518/1518 [==============================] - 0s 60us/step
[0.5337034053953269, 0.839262186931213]
In [61]:
test_GAP = Xception_model.get_GAP()
0.8150106851119865
In [0]:
stats_csv.add_stats("Xception Model", test_loss, test_accuracy, test_GAP)
Model Visualization :
In [38]:
Xception_model.plot_learning_curves(figures_dir)

Refinement

For refinement, I recreated VGG16 and Xception pre-trained models, but I changed the optimizer type for the top-model from stochastic gradient descent SGD to RMSProp first proposed by Geoffrey Hinton in his lectures Neural Networks for Machine Learning - Lecture 6a - Overview of mini-batch gradient descent.

Pretrained VGG16 Model (Refinement) :

Model Creation :
In [39]:
VGG16_model_refinement = PretrainedModel("VGG16_model_refinement",
                             applications.VGG16(include_top=False, weights='imagenet',pooling='avg'),
                             data_processor,
                             models_dir)
Model Training :
In [65]:
VGG16_model_refinement.predict_bottleneck_features()
7082/7082 [==============================] - 90s 13ms/step
1516/1516 [==============================] - 19s 13ms/step
1518/1518 [==============================] - 19s 13ms/step
In [40]:
VGG16_model_refinement.load_bottleneck_features()
In [41]:
VGG16_model_refinement.create_top_model(optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_14 (Dense)             (None, 256)               131328    
_________________________________________________________________
dense_15 (Dense)             (None, 128)               32896     
_________________________________________________________________
dropout_6 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_16 (Dense)             (None, 10)                1290      
=================================================================
Total params: 165,514
Trainable params: 165,514
Non-trainable params: 0
_________________________________________________________________
In [68]:
VGG16_model_refinement.train_top_model(epochs=1000, batch_size=4096)
Epoch 650/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.4343 - acc: 0.8448 - val_loss: 0.8874 - val_acc: 0.7210
Epoch 651/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.3705 - acc: 0.8707 - val_loss: 0.8456 - val_acc: 0.7381
Epoch 652/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.3462 - acc: 0.8812 - val_loss: 0.8936 - val_acc: 0.7216
Epoch 653/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.3747 - acc: 0.8700 - val_loss: 0.8987 - val_acc: 0.7236
Epoch 654/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3826 - acc: 0.8622 - val_loss: 0.8716 - val_acc: 0.7309
Epoch 655/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.3605 - acc: 0.8718 - val_loss: 0.8574 - val_acc: 0.7375
Epoch 656/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.3456 - acc: 0.8815 - val_loss: 0.8634 - val_acc: 0.7355
Epoch 657/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3617 - acc: 0.8739 - val_loss: 0.8842 - val_acc: 0.7315
Epoch 658/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3689 - acc: 0.8702 - val_loss: 0.8941 - val_acc: 0.7282
Epoch 659/1000
7082/7082 [==============================] - 0s 5us/step - loss: 0.3709 - acc: 0.8748 - val_loss: 0.8870 - val_acc: 0.7361
Epoch 660/1000
7082/7082 [==============================] - 0s 5us/step - loss: 0.3586 - acc: 0.8776 - val_loss: 0.8744 - val_acc: 0.7342
Epoch 661/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3441 - acc: 0.8824 - val_loss: 0.8578 - val_acc: 0.7342
Epoch 662/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3372 - acc: 0.8846 - val_loss: 0.8745 - val_acc: 0.7164
Epoch 663/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3396 - acc: 0.8848 - val_loss: 0.9106 - val_acc: 0.7131
Epoch 664/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3792 - acc: 0.8656 - val_loss: 0.9435 - val_acc: 0.7038
Epoch 665/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3917 - acc: 0.8596 - val_loss: 0.8488 - val_acc: 0.7368
Epoch 666/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3471 - acc: 0.8776 - val_loss: 0.8708 - val_acc: 0.7421
Epoch 667/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3714 - acc: 0.8690 - val_loss: 0.8581 - val_acc: 0.7427
Epoch 668/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3547 - acc: 0.8774 - val_loss: 0.8942 - val_acc: 0.7296
Epoch 669/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3651 - acc: 0.8716 - val_loss: 0.9116 - val_acc: 0.7203
Epoch 670/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3807 - acc: 0.8687 - val_loss: 0.9020 - val_acc: 0.7203
Epoch 671/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3660 - acc: 0.8738 - val_loss: 0.8725 - val_acc: 0.7434
Epoch 672/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3380 - acc: 0.8820 - val_loss: 0.8739 - val_acc: 0.7427
Epoch 673/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3558 - acc: 0.8756 - val_loss: 0.9121 - val_acc: 0.7216
Epoch 674/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3797 - acc: 0.8625 - val_loss: 0.8583 - val_acc: 0.7309
Epoch 675/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3280 - acc: 0.8870 - val_loss: 0.8637 - val_acc: 0.7282
Epoch 676/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3277 - acc: 0.8876 - val_loss: 0.9196 - val_acc: 0.7190
Epoch 677/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3687 - acc: 0.8670 - val_loss: 0.9123 - val_acc: 0.7249
Epoch 678/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3489 - acc: 0.8791 - val_loss: 0.8901 - val_acc: 0.7414
Epoch 679/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3387 - acc: 0.8829 - val_loss: 0.9321 - val_acc: 0.7342
Epoch 680/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3797 - acc: 0.8656 - val_loss: 0.9427 - val_acc: 0.7348
Epoch 681/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.4276 - acc: 0.8498 - val_loss: 0.8802 - val_acc: 0.7421
Epoch 682/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3454 - acc: 0.8822 - val_loss: 0.8641 - val_acc: 0.7414
Epoch 683/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3277 - acc: 0.8873 - val_loss: 0.9224 - val_acc: 0.7117
Epoch 684/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3559 - acc: 0.8732 - val_loss: 0.8647 - val_acc: 0.7328
Epoch 685/1000
7082/7082 [==============================] - 0s 3us/step - loss: 0.2978 - acc: 0.8988 - val_loss: 0.8259 - val_acc: 0.7474
Epoch 686/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.2851 - acc: 0.9026 - val_loss: 0.9047 - val_acc: 0.7322
Epoch 687/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3634 - acc: 0.8707 - val_loss: 0.9717 - val_acc: 0.7157
Epoch 688/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3894 - acc: 0.8652 - val_loss: 0.8548 - val_acc: 0.7513
Epoch 689/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3166 - acc: 0.8896 - val_loss: 0.8970 - val_acc: 0.7381
Epoch 690/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3450 - acc: 0.8766 - val_loss: 0.9168 - val_acc: 0.7269
Epoch 691/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3643 - acc: 0.8742 - val_loss: 0.9116 - val_acc: 0.7276
Epoch 692/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3478 - acc: 0.8836 - val_loss: 0.9331 - val_acc: 0.7256
Epoch 693/1000
7082/7082 [==============================] - 0s 4us/step - loss: 0.3703 - acc: 0.8657 - val_loss: 0.9043 - val_acc: 0.7309
Epoch 00693: early stopping
In [ ]:
VGG16_model_refinement.save_top_model_history()
In [42]:
VGG16_model_refinement.load_top_model_history()
In [0]:
 VGG16_model_refinement.load_top_model_weights()
Model Testing :
In [72]:
[test_loss, test_accuracy] = VGG16_model_refinement.test_top_model()
1518/1518 [==============================] - 0s 65us/step
[0.7469997773135918, 0.7733860340200079]
In [73]:
test_GAP = VGG16_model_refinement.get_GAP()
0.7329643747589238
In [0]:
stats_csv.add_stats("VGG16 Model(Refined)", test_loss, test_accuracy, test_GAP)
Model Visualization :
In [43]:
VGG16_model_refinement.plot_learning_curves(figures_dir)

Pretrained Xception Model (Refinement):

Model Creation :
In [44]:
Xception_model_refinement = PretrainedModel("Xception_model_refinement",
                             applications.Xception(include_top=False, weights='imagenet',pooling='avg'),
                             data_processor,
                             models_dir)
Model Training :
In [77]:
Xception_model_refinement.predict_bottleneck_features()
7082/7082 [==============================] - 120s 17ms/step
1516/1516 [==============================] - 26s 17ms/step
1518/1518 [==============================] - 26s 17ms/step
In [45]:
Xception_model_refinement.load_bottleneck_features()
In [46]:
Xception_model_refinement.create_top_model(optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0))
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_17 (Dense)             (None, 256)               524544    
_________________________________________________________________
dense_18 (Dense)             (None, 128)               32896     
_________________________________________________________________
dropout_7 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_19 (Dense)             (None, 10)                1290      
=================================================================
Total params: 558,730
Trainable params: 558,730
Non-trainable params: 0
_________________________________________________________________
In [ ]:
 
In [79]:
Xception_model_refinement.train_top_model(epochs=1000, batch_size=4096)
Epoch 248/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.1048 - acc: 0.9633 - val_loss: 1.5209 - val_acc: 0.7177
Epoch 249/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.6212 - acc: 0.8581 - val_loss: 1.0347 - val_acc: 0.7619
Epoch 250/1000
7082/7082 [==============================] - 0s 11us/step - loss: 0.1703 - acc: 0.9374 - val_loss: 0.7527 - val_acc: 0.8272
Epoch 251/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0565 - acc: 0.9876 - val_loss: 0.6404 - val_acc: 0.8437
Epoch 252/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0249 - acc: 0.9973 - val_loss: 0.6543 - val_acc: 0.8443
Epoch 253/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0206 - acc: 0.9976 - val_loss: 0.6707 - val_acc: 0.8450
Epoch 254/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0181 - acc: 0.9982 - val_loss: 0.6836 - val_acc: 0.8450
Epoch 255/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0163 - acc: 0.9989 - val_loss: 0.6951 - val_acc: 0.8430
Epoch 256/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0147 - acc: 0.9987 - val_loss: 0.7059 - val_acc: 0.8423
Epoch 257/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0146 - acc: 0.9982 - val_loss: 0.7152 - val_acc: 0.8423
Epoch 258/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0136 - acc: 0.9976 - val_loss: 0.7238 - val_acc: 0.8410
Epoch 259/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0133 - acc: 0.9987 - val_loss: 0.7320 - val_acc: 0.8384
Epoch 260/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0119 - acc: 0.9994 - val_loss: 0.7378 - val_acc: 0.8437
Epoch 261/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0122 - acc: 0.9989 - val_loss: 0.7450 - val_acc: 0.8410
Epoch 262/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0108 - acc: 0.9994 - val_loss: 0.7500 - val_acc: 0.8410
Epoch 263/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0114 - acc: 0.9986 - val_loss: 0.7521 - val_acc: 0.8397
Epoch 264/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0123 - acc: 0.9982 - val_loss: 0.7853 - val_acc: 0.8265
Epoch 265/1000
7082/7082 [==============================] - 0s 12us/step - loss: 0.0223 - acc: 0.9939 - val_loss: 1.1766 - val_acc: 0.7573
Epoch 266/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.5540 - acc: 0.8981 - val_loss: 0.8984 - val_acc: 0.7810
Epoch 267/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.1175 - acc: 0.9569 - val_loss: 0.7057 - val_acc: 0.8232
Epoch 268/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0342 - acc: 0.9918 - val_loss: 0.6717 - val_acc: 0.8371
Epoch 269/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0154 - acc: 0.9983 - val_loss: 0.6845 - val_acc: 0.8404
Epoch 00269: early stopping
In [ ]:
Xception_model_refinement.save_top_model_history()
In [47]:
Xception_model_refinement.load_top_model_history()
In [0]:
 Xception_model_refinement.load_top_model_weights()
Model Testing :
In [82]:
[test_loss, test_accuracy] = Xception_model_refinement.test_top_model()
1518/1518 [==============================] - 0s 67us/step
[0.5863029530397046, 0.8491436098561143]
In [83]:
test_GAP = Xception_model_refinement.get_GAP()
0.8267506001621842
In [0]:
stats_csv.add_stats("Xception Model(Refined)", test_loss, test_accuracy, test_GAP)
Model Visualization :
In [48]:
Xception_model_refinement.plot_learning_curves(figures_dir)

IV. Results

Model Evaluation and Validation

During model development, I used a validation set to evaluate the model while training. The validation set accuracy and loss scores were the basis for selecting the best top-model weights.

The final solution model chosen is Xception Model with the RMSprop optimizer. It was selected because it had the best performance on test accuracy and GAP scores. This model's performance is satisfactory given the nature and difficulty of the landmark recognition problem.

The following list describes our final solution model :

  • pre-trained Xception model without its top layers (include_top=False) and with average pooling mode for feature extraction (pooling='avg')
  • The input image size to the model is $240\times 240$ pixels.
  • A fully connected top model that has three Dense layers.
  • RMSProp optimizer. with learning rate lr=0.001

Sensitivity Analysis

To validate the robustness of the solution model, We will train the model with a different input image size of $150\times 150$.

Data Preprocessing

Trying different image input shape 150x150

In [0]:
evaluation_input_shape = (150, 150, 3)
In [87]:
evaluation_data_processor = DataProcessor(
    evaluation_input_shape, 
    batch_size, 
    train_dir, 
    validation_dir,
    test_dir)
Found 7082 images belonging to 10 classes.
Found 1516 images belonging to 10 classes.
Found 1518 images belonging to 10 classes.
Model Creation :
In [0]:
evaluation_model = PretrainedModel("evaluation_model",
                             applications.Xception(include_top=False, weights='imagenet',pooling='avg'),
                             evaluation_data_processor,
                             models_dir)
Model Training :
In [89]:
evaluation_model.predict_bottleneck_features()
7082/7082 [==============================] - 104s 15ms/step
1516/1516 [==============================] - 22s 14ms/step
1518/1518 [==============================] - 22s 14ms/step
In [0]:
evaluation_model.load_bottleneck_features()
In [90]:
evaluation_model.create_top_model(optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0))
Model: "sequential_11"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_31 (Dense)             (None, 256)               524544    
_________________________________________________________________
dense_32 (Dense)             (None, 128)               32896     
_________________________________________________________________
dropout_11 (Dropout)         (None, 128)               0         
_________________________________________________________________
dense_33 (Dense)             (None, 10)                1290      
=================================================================
Total params: 558,730
Trainable params: 558,730
Non-trainable params: 0
_________________________________________________________________
In [91]:
evaluation_model.train_top_model(epochs=1000, batch_size=4096)
Epoch 240/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0158 - acc: 0.9992 - val_loss: 1.0012 - val_acc: 0.7817
Epoch 241/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0161 - acc: 0.9993 - val_loss: 1.0117 - val_acc: 0.7836
Epoch 242/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0149 - acc: 0.9989 - val_loss: 1.0397 - val_acc: 0.7843
Epoch 243/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0143 - acc: 0.9987 - val_loss: 1.0399 - val_acc: 0.7856
Epoch 244/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0148 - acc: 0.9987 - val_loss: 1.0555 - val_acc: 0.7830
Epoch 245/1000
7082/7082 [==============================] - 0s 11us/step - loss: 0.0143 - acc: 0.9983 - val_loss: 1.0668 - val_acc: 0.7790
Epoch 246/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0141 - acc: 0.9996 - val_loss: 1.0771 - val_acc: 0.7823
Epoch 247/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0193 - acc: 0.9972 - val_loss: 1.3540 - val_acc: 0.7401
Epoch 248/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.9216 - acc: 0.8234 - val_loss: 2.3202 - val_acc: 0.6623
Epoch 249/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.8029 - acc: 0.8241 - val_loss: 1.2489 - val_acc: 0.7460
Epoch 250/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.1047 - acc: 0.9670 - val_loss: 0.9615 - val_acc: 0.7823
Epoch 251/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0256 - acc: 0.9976 - val_loss: 0.9684 - val_acc: 0.7810
Epoch 252/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0221 - acc: 0.9983 - val_loss: 0.9785 - val_acc: 0.7830
Epoch 253/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0188 - acc: 0.9992 - val_loss: 0.9879 - val_acc: 0.7836
Epoch 254/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0184 - acc: 0.9986 - val_loss: 0.9960 - val_acc: 0.7856
Epoch 255/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0169 - acc: 0.9990 - val_loss: 1.0078 - val_acc: 0.7850
Epoch 256/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0161 - acc: 0.9987 - val_loss: 1.0166 - val_acc: 0.7869
Epoch 257/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0154 - acc: 0.9992 - val_loss: 1.0258 - val_acc: 0.7883
Epoch 258/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0147 - acc: 0.9987 - val_loss: 1.0352 - val_acc: 0.7876
Epoch 259/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0140 - acc: 0.9993 - val_loss: 1.0399 - val_acc: 0.7883
Epoch 260/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0127 - acc: 0.9994 - val_loss: 1.0491 - val_acc: 0.7863
Epoch 261/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0128 - acc: 0.9989 - val_loss: 1.0535 - val_acc: 0.7876
Epoch 262/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0121 - acc: 0.9990 - val_loss: 1.0644 - val_acc: 0.7856
Epoch 263/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0116 - acc: 0.9990 - val_loss: 1.0731 - val_acc: 0.7869
Epoch 264/1000
7082/7082 [==============================] - 0s 10us/step - loss: 0.0114 - acc: 0.9990 - val_loss: 1.0791 - val_acc: 0.7883
Epoch 265/1000
7082/7082 [==============================] - 0s 9us/step - loss: 0.0110 - acc: 0.9990 - val_loss: 1.0888 - val_acc: 0.7876
Epoch 00265: early stopping
In [0]:
evaluation_model.save_top_model_history()
#evaluation_model.load_top_model_history()
In [0]:
 evaluation_model.load_top_model_weights()
Model Testing :
In [94]:
[test_loss, test_accuracy] = evaluation_model.test_top_model()
1518/1518 [==============================] - 0s 80us/step
[0.8985504357239945, 0.7793148879320096]
In [95]:
test_GAP = evaluation_model.get_GAP()
0.7417914545384348
In [0]:
#stats_csv.add_stats("Solution Model Validation", test_loss, test_accuracy, test_GAP)
Model Visualization :
In [97]:
evaluation_model.plot_learning_curves(figures_dir)

The test loss score was 0.8985, the accuracy score was 77.93\%, and the GAP score was 74.18\%. The performance is slightly less but not very far from using an input size of $240\times 240$.

Justification

The final solution model has 84.91% test accuracy score and 82.68% GAP score. Its performance is significantly higher than the benchmark model which had 62.29% test accuracy score and 55.85% GAP score.

V. Conclusion

Free-Form Visualization

I will test my final solution model on three unseen landmark images randomly downloaded from the web for Jami Masjid in Feroz Shah Kolta, Cathedral of the Apostles Peter and Paul in Kazan and Golden Gate Bridge

Visualization of predicting images using solution model :

In [0]:
from tqdm import tqdm
from sklearn.datasets import load_files
In [0]:
def extract_VGG16(VGG16model, tensor):
	from keras.applications.vgg16 import VGG16, preprocess_input
	return VGG16model.predict(preprocess_input(tensor))
In [0]:
def extract_Xception(Xceptionmodel, tensor):
	from keras.applications.xception import Xception, preprocess_input
	return Xceptionmodel.predict(preprocess_input(tensor))

Here we assign our best solution model to the variable solution_model and preprocessor function to extract_bottleneck_features variable

In [0]:
solution_model = Xception_model_refinement
In [0]:
extract_bottleneck_features = extract_Xception

In [0]:
def path_to_tensor(img_path):
    # loads RGB image as PIL.Image.Image type
    img = load_img(img_path, target_size=(224, 224))
    # convert PIL.Image.Image type to 3D tensor with shape (224, 224, 3)
    x = img_to_array(img)
    # convert 3D tensor to 4D tensor with shape (1, 224, 224, 3) and return 4D tensor
    return np.expand_dims(x, axis=0)

def paths_to_tensor(img_paths):
    list_of_tensors = [path_to_tensor(img_path) for img_path in tqdm(img_paths)]
    return np.vstack(list_of_tensors)
In [0]:
# Loading test image paths
test_data = load_files(test_images_dir)
test_images_paths = np.array(test_data['filenames'])
In [107]:
# Getting prediciton labels using solution model :
bottleneck_features = extract_bottleneck_features(solution_model.pretrained_model, paths_to_tensor(test_images_paths))
predicted_classes = solution_model.top_model.predict_classes(bottleneck_features)
predicted_labels = np.array(landmarks_list)[predicted_classes]
  0%|          | 0/3 [00:00<?, ?it/s]
100%|██████████| 3/3 [00:00<00:00, 49.95it/s]

The following run shows that the solution model does a good job classifying the three images correctly.

In [109]:
# Plot test images and their predicted labels :
columns = len(predicted_labels)
rows = 1
fig = plt.figure(figsize=(20,20))
for i, image_path in enumerate(test_images_paths):
    fig.add_subplot(rows, columns, i+1)
    plt.imshow(load_img(image_path))
    plt.title(predicted_labels[i])
plt.savefig(path.join(figures_dir, "visualization_predicted_images.pdf"), bbox_inches = 'tight')
plt.show()

Visualization of similar images from training dataset

Visually similar images from the training dataset that the model was trained on are shown below :

In [0]:
def get_train_image_path(image_id, label):
  return path.join(train_dir, label,  image_id + '.jpg')
In [0]:
similar_train_landmarks_ids =['Kazan','Feroz_Shah_Kotla', 'Golden_Gate_Bridge']
In [0]:
similar_train_images_ids = np.array([
                                    ['3e222ad7d1469deb', '33e8a9c8e96f20eb', '1586d22396244714'],
                                    ['1e33638e9fb39b0d', '2f7b8d029e1402b4', '1e4569e97ea7ade0'],
                                    ['5cd678ac220edb5a', '8c188787c993afba', '8e471206a5892d58'],
                                    ])
In [113]:
rows = similar_train_images_ids.shape[0]
columns = similar_train_images_ids.shape[1]
fig = plt.figure(figsize=(20,20))

for i, landmark in enumerate(similar_train_images_ids):
  for j, image in enumerate(landmark):
      image_path = get_train_image_path(str(image), str(similar_train_landmarks_ids[i]))
      fig.add_subplot(rows, columns, columns*i + j + 1)
      plt.imshow(load_img(image_path))
      plt.title(similar_train_landmarks_ids[i])

plt.savefig(path.join(figures_dir, "visualization_training_samples.pdf"), bbox_inches = 'tight')
plt.show()

Reflection

The entire end-to-end problem solution can be summarized as the following :

  1. A challenge problem and an available public dataset were found.
  2. A suitable metric was found and implemented.
  3. The data was downloaded and split into training, validation and testing sets.
  4. The data was prepared and preprocessed to be used as input for the classification model.
  5. A simple convolutional neural network benchmark model was implemented and tested.
  6. Transfer-learning solution models VGG16 and Xception were implemented, refined and tested. The extraction of bottleneck features was done for each model. Then, a solution model was selected based on performance metrics.
  7. Sensitivity analysis was conducted on the solution model using a different resolution of input images.
  8. The solution model classifier was tested on unseen images downloaded randomly from the web.

An interesting aspect of this project was that transfer-learning models achieved great performance in short training time. Their performance was better than the benchmark model that took a much longer time to train.

The challenging aspect of the project was the selection of classes and the extraction of a balanced subset dataset for the project from the large publicly available dataset. Another challenge was the implementation of the GAP performance metric and obtaining the correct input vectors.

Convolutional neural networks in general and transfer-learning techniques, in particular, are the best for image classification problems (up to current date) as we have seen in this project implementation. They are highly recommended for such problems and similar problems.

Improvement

For the improvement of our solution model, my suggestions are the following :

  1. Allowing the model to train for longer times on more powerful hardware with capable GPUs. That would speed up experimenting with and testing different solution models in shorter times.
  2. Another possible improvement is collecting a larger dataset of landmarks, training the model on a larger count of images and augmenting the available data to increase the count of images per each class.
  3. In this project, we froze the original pre-trained networks’ weights and focused on training the top-model that we created (which took bottleneck features as input). We can try fine-tuning each of the original pre-trained models’ weights to achieve a better feature extraction from the input.
  4. Proper classification of non-landmark images and the ability to recognize them should be implemented.

Copying output data from colab virtual machine to google drive :

In [114]:
cd /
/
In [115]:
%%shell
tar cvzf stats.tar.gz docs/
tar cvzf models.tar.gz models/
cp -v stats.tar.gz /gdrive/My\ Drive/shared/Udacity_MLND_capstone_dataset/
cp -v models.tar.gz /gdrive/My\ Drive/shared/Udacity_MLND_capstone_dataset/
docs/
docs/stats/
docs/stats/train_freqs.csv
docs/stats/validation_freqs.csv
docs/stats/selected_landmarks.csv
docs/stats/stats.csv
docs/stats/test_freqs.csv
docs/img/
docs/img/Xception_model_refinement_metrics.pdf
docs/img/benchmark_model_metrics.pdf
docs/img/augmented_image_transformations.pdf
docs/img/VGG16_model_refinement_metrics.pdf
docs/img/VGG16_model_metrics.pdf
docs/img/augmented_image_original.pdf
docs/img/evaluation_model_metrics.pdf
docs/img/visualization_predicted_images.pdf
docs/img/dataset_hbar_plot.pdf
docs/img/Xception_model_metrics.pdf
docs/img/visualization_training_samples.pdf
models/
models/VGG16_model_top-model_history.pickle
models/Xception_model_bottleneck_features_train.npz
models/VGG16_model_refinement_bottleneck_features_train.npz
models/VGG16_model_bottleneck_features_validation.npz
models/VGG16_model_bottleneck_features_test.npz
models/Xception_model_refinement_bottleneck_features_train.npz
models/Xception_model_top-model_history.pickle
models/VGG16_model_refinement_bottleneck_features_test.npz
models/VGG16_model_refinement_top-model_weights.hdf5
models/Xception_model_bottleneck_features_validation.npz
models/evaluation_model_bottleneck_features_train.npz
models/benchmark-model_weights.hdf5
models/Xception_model_refinement_top-model_weights.hdf5
models/Xception_model_top-model_weights.hdf5
models/Xception_model_refinement_bottleneck_features_validation.npz
models/evaluation_model_bottleneck_features_validation.npz
models/Xception_model_refinement_top-model_history.pickle
models/evaluation_model_bottleneck_features_test.npz
models/VGG16_model_refinement_bottleneck_features_validation.npz
models/benchmark-model_history.pickle
models/Xception_model_refinement_bottleneck_features_test.npz
models/VGG16_model_top-model_weights.hdf5
models/VGG16_model_bottleneck_features_train.npz
models/evaluation_model_top-model_history.pickle
models/Xception_model_bottleneck_features_test.npz
models/evaluation_model_top-model_weights.hdf5
models/VGG16_model_refinement_top-model_history.pickle
Out[115]: